YQL talk from NYTimes Hackday 2010

1. YQL
SELECT * FROM Internet
Developer Talk Token Unicorn

2. Derek Gathright

3. Derek Gathright
• Engineer @ Yahoo

4. Derek Gathright
• Engineer @ Yahoo
• Tweet from @derek

5. Derek Gathright
• Engineer @ Yahoo
• Tweet from @derek
• Blog @ http://derekville.net

6. Derek Gathright
• Engineer @ Yahoo
• Tweet from @derek
• Blog @ http://derekville.net
• Everything else @ http://derek.io

7. On December 4th, 1995

8. On December 4th, 1995
“Netscape and Sun announce JavaScript, a
cross-platform object scripting language”
http://web.archive.org/web/20070916144913/http://wp.netscape.com/newsref/pr/newsrelease67.html

9. How big is the Web?

10. GINOR
MOUS!

11. The Internet: 1995
http://www.jevans.com/pubnetmap.html

12. The Internet: 2003
http://www.opte.org/maps/

13. The Internet: 2007
http://xkcd.com/256/

14. The Internet: 2010
http://xkcd.com/802/

15. It’s impossible to
measure the size of the
web because it is
constantly changing,
growing, and morphing.

16. Every second:
Twitter gets 600 new tweets

17. Every minute:
YouTube +35 hours of video

18. Every month: Facebook gets
2.5 billion new photos

19. Yeah, lots of it is garbage

20. But there's still a ton of
interesting stuff out there.

21. So how do you access it,
programmatically?

22. It is easy enough to
‘scrape’ the web (using
cURL, wget, etc...), but
how do you parse it?

23. It is easy enough to
‘scrape’ the web (using
cURL, wget, etc...), but
how do you parse it?
XPath + DOM Traversal = Yay!

24. It is easy enough to
‘scrape’ the web (using
cURL, wget, etc...), but
how do you parse it?
XPath + DOM Traversal = Yay!
Regular Expressions = Double Yay!

25. It is easy enough to
‘scrape’ the web (using
cURL, wget, etc...), but
how do you parse it?

26. “If a regular expression
is longer than 2 inches,
find another method”
- Douglas Crockford

28. 4.4 lbs &1,368 pages.
No thanks!

29. The Point?
The Web has a ton of
data, but no easy,
hackday-able way to
access it.

30. THE WEB
NEEDS
AN API!

31. APIs are awesome.
You get (mostly) whatever data
you want in (mostly) whatever
format you want and in a (mostly)
easy to parse structure.

32. Example...
http://api.twitter.com/1/users/
show.json?screen_name=derek

33. Companies discovered that if you build
APIs, developers will come in droves

34. And it saves you from having to
dance around on stage like a monkey
(if you are confused, Google “monkey dance”)

35. Yahoo, Google,
Facebook, Twitter,
Microsoft, NY Times, ...
Most web companies
offer APIs now days.

36. As neat as they are,
they are imperfect.
Why?

37. As neat as they are,
they are imperfect.
Why?
You have to read
documentation to use
them.

38. As neat as they are,
they are imperfect.
Why?
You have to read
documentation to use
them.
BOOOOOOO!!!!

39. We're developers,
we're lazy,
and we want to build stuff,
NOW!

40. Yahoo invented a
solution to this
problem...

42. YQL!

43. "Yahoo! Query Language is an
expressive SQL-like language that
lets you query, filter, and join data
across Web services.
With YQL, apps run faster with
fewer lines of code and a smaller
network footprint."
http://developer.yahoo.com/yql/

44. YQL is...
• RESTful
• Scaleable
• Customizable
• ... and lots of other “ables”

45. How do you use it?

46. How do you use it?
$format = “json”; // or xml;

47. How do you use it?
$format = “json”; // or xml;
$base = “http://query.yahooapis.com/v1/public/yql”;

48. How do you use it?
$format = “json”; // or xml;
$base = “http://query.yahooapis.com/v1/public/yql”;
$url = “{$base}?q={$yql_query}&format={$format}”;

49. How do you use it?
$format = “json”; // or xml;
$base = “http://query.yahooapis.com/v1/public/yql”;
$url = “{$base}?q={$yql_query}&format={$format}”;
$json_string = goGetIt($url); // likely a curl() call

50. How do you use it?
$format = “json”; // or xml;
$base = “http://query.yahooapis.com/v1/public/yql”;
$url = “{$base}?q={$yql_query}&format={$format}”;
$json_string = goGetIt($url); // likely a curl() call
$data = json_decode($json);

51. How do you use it?
$format = “json”; // or xml;
$base = “http://query.yahooapis.com/v1/public/yql”;
$url = “{$base}?q={$yql_query}&format={$format}”;
$json_string = goGetIt($url); // likely a curl() call
$data = json_decode($json);
Or use any of the libraries written for
your favorite language/framework

52. YQL Queries
SELECT {fields}
FROM {table}
WHERE {conditions}

53. SELECT * FROM
weather.forecast
WHERE
location=90210

54. SELECT * FROM
data.html.cssselect
WHERE
url=“http://yahoo.com”
AND css=“body a”;

55. SELECT height,width,url
FROM search.images
WHERE
query=“kitteh” AND
mimetype LIKE “%jpeg%”

56. SELECT * FROM
google.search
WHERE
q=“pizza”

57. SELECT status.text FROM
twitter.user.timeline
WHERE
screen_name=“derek”

58. SELECT * FROM
foursquare.history
WHERE
username=“foo” AND
password=“bar”

59. SELECT * FROM rss WHERE
url IN (SELECT title FROM
atom WHERE url="http://
spreadsheets.google.com/feeds/list/
pg_T0Mv3iBwIJoc82J1G8aQ/od6/public/
basic")
LIMIT 10 |
unique(field="title")

60. Where’s the magic?

61. Data Tables
13 categories & counting, including...
• Geo • Social
• Flickr • Upcoming
• Local • Weather
• Maps • Yahoo (Search)
• Meme • YMail
• Music • YQL (Storage)

62. Open Data Tables
900+ community contributed tables in
hundreds of categories, including...
Amazon • Netflix
• Craigslist • NY Times
• Facebook • SimpleGeo
• Foursquare • SPARQL
• Google • Twitter
• HackerNews • Wordpress
• LastFM • YouTube

64. google.search

65. google.search
http://www.datatables.org/google/google.search.xml

66. google.search
http://www.datatables.org/google/google.search.xml
twitter.user.timeline

67. google.search
http://www.datatables.org/google/google.search.xml
twitter.user.timeline
http://www.datatables.org/twitter/twitter.user.timeline.xml

68. google.search
http://www.datatables.org/google/google.search.xml
twitter.user.timeline
http://www.datatables.org/twitter/twitter.user.timeline.xml
foursquare.history

69. google.search
http://www.datatables.org/google/google.search.xml
twitter.user.timeline
http://www.datatables.org/twitter/twitter.user.timeline.xml
foursquare.history
http://www.datatables.org/foursquare/history.xml

70. http://www.datatables.org/craigslist/craigslist.search.xml

71. http://www.datatables.org/craigslist/craigslist.search.xml

72. http://www.datatables.org/craigslist/craigslist.search.xml

73. http://www.datatables.org/craigslist/craigslist.search.xml

74. &query={query}
http://www.datatables.org/craigslist/craigslist.search.xml

75. YQL != Voodoo Magic
It is just rewriting a YQL
query into one (or many)
HTTP calls for you.

76. USE "http://www.datatables.org/nyt/nyt.bestsellers.xml"
AS nyt.bestsellers;
USE "https://github.com/gcb/yql.opentable/raw/master/
text.concat.xml"
AS text.concat;
SELECT text FROM text.concat
WHERE text.key1 = "http://www.amazon.com/dp/" AND
(text.key2) IN (
SELECT isbns.isbn.isbn10
FROM nyt.bestsellers
WHERE apikey='yourAPIKey'
);
// Generates strings like “http://www.amazon.com/dp/031603617X”

77. USE "https://github.com/gcb/yql.opentable/raw/master/
text.concat.xml"
AS text.concat;

78. https://github.com/gcb/yql.opentable/raw/master/text.concat.xml

79. https://github.com/gcb/yql.opentable/raw/master/text.concat.xml

80. <execute>
• Execute arbitrary JavaScript in Rhino (a JS engine)
• E4X Support (XML literals in JS)
• Speak protocols and handle authentication;
Basic auth, OAuth, XAuth, XMLRPC, ...
• Best feature? View-source!

81. Summary
• YQL is very useful for...
• Scraping
• Creating an API where one doesn’t exist
• Converting XML -> JSON, & vice-versa
• JSONP for JS-only apps
• Many HTTP requests -> single HTTP request
• Server-side JS processing

82. NY Times Data Tables
• nyt.article.search • nyt.newswire
• nyt.bestsellers • nyt.people.activities
• nyt.bestsellers.history • nyt.people.followers
• nyt.bestsellers.search • nyt.people.following
• nyt.movies.critics • nyt.people.newsfeed
• nyt.movies.picks • nyt.people.profiles
• nyt.movies.reviews • nyt.people.users

83. Get started @
http://developer.yahoo.com/yql
Thanks!
Questions? Find, Tweet, or Email me.
@derek or drg@yahoo-inc.com